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Abstract 

The  Medicare  Patient  Safety  Monitoring  System  (MPSMS)  is  a  national 
surveillance  project  aimed  at  identifying  the  rates  of  specific  adverse  events 
within  the  Medicare  population.  Created  under  the  auspices  of  the  Department  of 
Health  and  Human  Services’  Patient  Safety  Task  Force,  this  surveillance  system 
identifies  adverse  events  from  randomly  selected  inpatient  Medicare  discharges 
and  administrative  data.  This  system  is  immense  in  scope  and  provides  national 
rates  of  adverse  events  by  employing  explicit  review  criteria  for  all  patient  safety 
topics.  The  MPSMS  explicit  review  is  a  patient-centered  process  focusing  on 
patient  hann  rather  than  provider  or  system  error,  and  has  the  following  features: 
nonnalized  inter-rater  reliability;  lower  cost-per-chart  reviewed  than  the 
traditional  clinical  expert-based  implicit  review;  and  the  potential  for  comparative 
analysis  across  time  and  health  care  systems.  A  limitation  to  our  approach  is  the 
reliance  on  administrative  data  to  complete  any  post-discharge  surveillance 
required.  This  paper  explores  the  precepts  behind  the  MPSMS  review  process. 
Three  principles — intent,  relevance,  and  transparency — describe  the  conceptual 
underpinnings  for  our  approach. 


Introduction 

Patient  safety,  while  not  a  new  topic,  recently  has  experienced  a  renewed 
sense  of  urgency  in  public  health  discourse.  This  urgency  is,  in  part,  precipitated 
by  the  confluence  of  market  pressures  to  demonstrate  value  in  a  health  care 
system  increasing  in  technological  sophistication  yet  lacking  the  commensurate 
capacity  to  measure  and,  in  turn,  reduce  the  rate  of  fundamental  defects  as 
measured  by  patient  harm.  This  point  was  brought  into  sharp  focus  in  the  2000 
Institute  of  Medicine  (IOM)  report,  To  Err  Is  Human:  Building  a  Safer  Health 
System}  The  IOM  reported  that  between  44,000  and  98,000  patients  die  each  year 
in  the  United  States,  as  a  result  of  their  interaction  with  health  care  delivery 
systems.  Two  byproducts  of  that  report  are  invigorated  programs  to  measure  and 
improve  the  safety  of  those  systems.  A  surveillance  system,  the  MPSMS,  is  one 
such  program  and  has  the  goal  of  measuring  the  rate  of  specific  inpatient  adverse 
events  within  the  Medicare  population. 
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Methods 

Sampling  strategy  and  record  abstraction 

The  hospital  records  from  an  annual  random  national  sampling  of  more  than 
40,000  Medicare  inpatient  fee-for-service  discharges  are  sent  to  two  Clinical  Data 
Abstraction  Centers  (CD AC).  At  the  CDACs,  trained  abstractors  processed  each 
clinical  chart  by  recording  predefined  chart  data.  Abstractor  training  for  this 
process  includes  didactic  instruction  and  documentation  regarding  concept 
definitions;  rules  of  precedence  regarding  where  in  the  chart  predefined 
information  should  be  extracted;  and  key  terminology  and  synonyms. 

Determination  of  inter-rater  reliability  and  accuracy  is  part  of  the  MPSMS 
Internal  Quality  Control  (IQC).  As  part  of  that  process,  every  month  40  randomly 
selected  abstracted  charts  are  exchanged  between  the  two  CDACs.  These  charts 
are  then  re-abstracted  and  the  aggregated  results  are  listed  (Table  1).  Adjudication 
of  results  in  which  there  is  a  discrepancy  between  abstracts  is  overseen  by 
members  of  the  MPSMS  contractor,  Qualidigm,  and  the  Connecticut  Quality 
Improvement  Organization  (QIO).  The  results  of  the  adjudication  are  then 
incorporated  into  the  CDAC  abstractor  training  and  instruction  documentation. 


Table  1:  Aggregated  MPSMS  reliability 


Completeness  measure 

99.5% 

Number  of  cases  produced 

40,620 

Number  of  cases  sampled 

480 

Inter-rater  reliability 

96.66% 

Overall  accuracy 

98.03% 

For  the  purposes  of  this  report,  the  completeness  measure  is  based  on  the 
actual  case  data  and  is  not  a  measure  of  validated  responses.  This  measure 
provides  information  about  the  average  completeness  of  a  case  in  the  IQC  sample. 
The  denominator  includes  all  variables  with  an  abstracted  response  for  all  original 
and  re-abstracted  cases.  The  numerator  includes  all  variables  with  a  response 
excluding  the  one,  unable  to  determine  (UTD).  The  percent  complete  is  the 
aggregate  average  of  all  cases. 

Inter-rater  reliability  is  calculated  as  the  raw  agreement  rate  between  the 
original  abstractor  and  the  re-abstractor.  For  example,  if  the  module  contains  100 
data  elements  and  the  abstractors  agree  on  90  of  them,  the  reliability  score  would 
be  90  percent.  The  inter-rater  reliability  is  the  aggregate  agreement  rate  across  all 
data  elements  in  all  cases  in  the  IQC  sample. 

Accuracy  is  calculated  as  the  raw  agreement  rate  of  both  the  original 
abstractor  and  the  re-abstractor  with  the  adjudicated  gold  standard  data.  The 
overall  accuracy  is  the  aggregate  agreement  rate  across  all  data  elements  in  all 
cases  in  the  IQC  sample. 
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The  final  data  source  for  the  MPSMS  is  the  CMS  administrative  data  set  for 
each  of  the  randomly  selected  inpatient  records.  This  data  set  contains  all  the 
information  from  beneficiary  Medicare  Part  A  claims,  Medicare  Part  B  claims, 
and  the  beneficiary  master  file.  These  files  contain  hospital  admission  and 
professional  component  procedure  and  diagnosis  codes.  The  beneficiary  master 
file  includes  socio-demographic  information.  The  value  of  the  administrative  data 
set  is  that  it  can  be  used  to  complement  information  garnered  for  an  individual 
inpatient  episode  of  care,  such  as  30-day  postoperative  mortality,  readmission,  or 
return  to  the  operating  room.  This  is  very  important  given  that  the  MPSMS  has  no 
capacity  to  obtain  additional  or  related  records  for  an  individual  case  beyond  the 
initial,  randomly  selected  hospital  chart. 

Definition  of  variables  and  explicit  criteria 

An  adverse  event  is  defined  as  an  unintended  harm,  injury,  or  loss  that  is  more 
likely  associated  with  a  patient’s  interaction  with  the  health  care  delivery  system 
than  from  an  attendant  disease  process.  The  MPSMS  identifies  adverse  events  in 
discrete  clinical  topics  only  and  there  is  no  attempt  to  identify  adverse  events 
beyond  the  list  of  MPSMS  topics.  An  MPSMS  topic  algorithm,  algorithm 
abstraction  instructions,  and  algorithm  definitions  wholly  define  an  adverse  event 
in  the  MPSMS.  The  first  year  of  abstraction  involved  data  collection  and  analysis 
for  adverse  events  in  the  following  topics:  bloodstream  infections;  central  venous 
catheters  (CVC);  total  hip  and  total  knee  replacement;  postoperative  pneumonia, 
postoperative  venous  thromboembolic  events;  postoperative  urinary  tract 
infection;  and  ventilator  associated  pneumonia.  All  chart  abstraction  is  performed 
at  CDACs  and  all  data  analysis  is  performed  at  the  Connecticut  Medicare  Quality 
Improvement  Organization,  Qualidigm. 

Once  proposed,  a  topic  is  vetted  through  a  process  that  first  includes  providing 
our  entire  technical  expert  panel  (TEP)  with  a  thorough  literature  review.  As  we 
compile  and  share  the  literature  review,  both  the  strengths  and  weaknesses  of  the 
potential  MPSMS  topic  are  discussed  with  the  TEP  and  other  national  experts. 
Especially  important  are  the  opinions  of  CMS's  federal  agency  TEP  partners, 
Centers  for  Disease  Control  and  Prevention  (CDC),  Food  and  Drug 
Administration  (FDA),  Agency  for  Healthcare  Research  and  Quality  (AHRQ), 
and  Department  of  Veterans  Affairs  (VA).  Throughout  the  process,  practical 
considerations  such  as  the  estimated  frequency  of  exposures  in  the  Medicare  fee- 
for-service  inpatient  population  and  the  richness  of  event  descriptors  (or  proxies) 
being  documented  in  the  record  are  considered.  If  a  topic  meets  all  reasonable 
standards  to  this  point,  the  MPSMS  team  at  Qualidigm  then  designs  preliminary 
technical  specifications  with  input  from  selected  TEP  subject  matter  experts. 

These  preliminary  technical  specifications  define  each  adverse  event  category; 
provide  specifications  of  numerator(s)  and  denominator(s);  list  inclusion  and 
exclusion  criteria;  and  define  data  sources  for  each  measure.  Specifically,  for  each 
measure,  the  preliminary  technical  specification  includes — 
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1 .  Exposure/adverse  event  (the  name  of  the  exposure/adverse  event). 

2.  Description  (a  description  of  the  exposure/adverse  event  measure). 

3.  Definition  (a  detailed  definition  of  the  exposure/adverse  event 
measure). 

4.  Adverse  event  (a  list  of  adverse  events  to  be  examined  for  each 
measure). 

A  list  of  critical  definitions  and  synonyms  is  complied,  concurrent  with  the 
algorithm  design.  The  algorithm  is  then  “alpha”  tested  with  nurse  reviewers  to 
identify  if  explicit  discriminators  within  the  algorithm  can  be  extracted  from  a 
sample.  Detailed  results  and  comments  from  this  review  are  analyzed,  and,  if 
necessary,  algorithm  improvements  exacted. 

The  results  of  alpha  testing  allow  for  refinement  of  the  data  collection  tool,  the 
tool  instructions,  and  analysis  algorithms.  The  “beta”  test  serves  as  the  next  level 
of  testing.  It  is  used  to  assess  the  validity  and  reliability  of  the  data  collection  tool 
variables  for  each  specific  measure. 

The  beta  test  includes  the  following  components: 

•  revisions  to  the  measure  algorithms; 

•  refinement  of  the  data  collection  tool  and  instructions  to  improve 
accuracy,  consistency  and  ease  of  abstraction; 

•  assessment  of  the  reliability  and  validity  of  the  abstractors  and 
abstracted  data  elements;  and 

•  completion  of  a  test  run  of  data  processing  and  reporting  methods. 

Specifically,  Beta  testing  involves — 

•  creating  gold  standard  records; 

•  validating  abstractors’  work  using  gold  standard  records; 

•  testing  abstractor-level  reliability; 

•  assuring  95  percent  inter-rater  reliability;  and 

•  performing  clinical  review  for  the  measure  exposures  and  associated 
adverse  events. 

Independent,  practicing  clinicians  are  employed  for  the  last  phase  of  beta 
testing,  clinical  review.  The  purpose  of  the  MPSMS  beta  clinical  review  process 
is  twofold: 

1 .  To  determine  if  practicing  clinicians,  using  their  clinical  judgment  and 
Qualidigm  and  CDAC  abstractors,  were  able  to  identify  the  same 
patients  as  being  exposed  to  a  specific  health  care  delivery  process, 
and  identify  the  same  patients  as  experiencing  specific  adverse  events 
associated  with  the  exposures. 


108 


Medicare  Patient  Safety  Surveillance 


2.  To  assess  the  clinician  reviewers’  degree  of  certainty  that  the  adverse 
events  identified  were  associated  with  the  exposure.  This  information 
will  be  used  in  the  overall  evaluation  of  the  effectiveness  of  the 
MPSMS  beta  abstraction  tool  and  algorithm  in  detecting  health  care 
exposures  and  associated  adverse  events. 

Each  topic  module  and  any  associated  adverse  event  rate  data  are  reviewed  at 
least  annually  by  the  TEP. 

Confidentiality 

MPSMS  is  a  quality  improvement  project.  Thus,  information  collected  is 
protected  by  Federal  law  against  disclosure  in  a  fonn  that  identifies  individuals  or 
providers,  and  against  discovery  or  subpoena  in  civil  actions.  This  project  also 
meets  all  the  requirements  of  Health  Insurance  Portability  and  Accountability  Act 
of  1996  (HIP AA). 

Discussion 

The  purpose  of  this  paper  is  to  present  the  philosophical  underpinnings  of  the 
MPSMS  as  a  new  paradigm  in  patient  safety  monitoring.  Beyond  the  uniqueness 
of  this  surveillance  system,  the  MPSMS  team  recognized  the  inherent  imperative 
that  those  who  conduct  patient  safety  surveillance  thoroughly  describe  not  only  a 
study’s  methodology  but  also  the  thought  processes  that  guided  the  methodology 
design  and  maintenance.  Because  results  of  national  patient  safety  surveillance 
studies  are  used  at  all  levels  of  health  care  services  delivery,  from  public  health 
policy  to  specific  provider  treatment  decisions,  a  descriptive  acknowledging  the 
study  gestalt  has  intrinsic  value.  To  that  end,  it  is  best  to  first  recognize  that  the 
MPSMS  relies  solely  on  retrospective  review  of  written  hospital  records  and 
insurance  claims.  In  that  regard,  the  MPSMS  remains  in  the  same  category  of 
patient  safety  surveillance  as  the  AHRQ  Patient  Safety  Indicators  (PSI)  and  the 
Harvard  Medical  Practice  Review  in  that  there  is  no  provision  to  acquire 
information  beyond  those  two  records. 

Still,  the  MPSMS  represents  a  new  paradigm.  Adverse  event  determinations, 
based  on  AHRQ  Patient  Safety  Indicators,  are  exclusively  derived  from  claims 
and  are  incapable  of  discriminating  events  by  exploiting  the  clinical  detail  of  the 
hospital  chart  not  reflected  in  an  insurance  claim.  Conversely,  patient  safety 
surveillance  as  performed  in  the  Harvard  medical  practice  study  is  limited  by 
“moderate  reliability  of  judgements  about  adverse  events.”2  The  Harvard 
approach  employs  structured  implicit  review  instruments.  Clinicians  are  “trained 
to  review  hospital  medical  records  and  give  their  opinion  on  the  occurrence  of 
adverse  events  and  the  quality  of  hospital  care  and  its  impact  on  patient 
outcomes,”3  MPSMS  the  new  surveillance  paradigm,  is  designed  by  CMS  to 
conduct  an  explicit  review  by  extracting  relevant  data  in  both  the  clinical  record 
and  the  administrative  claims  data  set  with  a  high  degree  of  inter-rater  reliability. 
A  well-defined  algorithm  is  then  applied  to  that  data  to  determine  a  rate  of 
adverse  events.  As  such,  the  MPSMS  methods,  algorithms,  and  definitions  may 
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be  employed  to  establish  rates  of  adverse  events  by  any  entity  with  access  to  a 
large  chart-based  data  source  (e.g.,  hospitals,  health  plans,  provider  networks,  and 
payers).  Before  any  such  employ,  it  is  essential  to  understand  the  fundamental 
precepts  behind  our  surveillance  project  design.  For  this,  we  find  it  most  useful  to 
describe  the  MPSMS  relative  to  three  coordinates  or  requirement  planes:  intent, 
relevance,  and  transparency. 

Intent 

Intent  answers  the  broader  question:  “For  what  purpose  will  the  infonnation 
be  used?”  To  fully  answer,  it  is  best  first  to  define  how  the  MPSMS  fits  into  the 
larger  Centers  for  Medicare  and  Medicaid  Services  (CMS)  Quality  Improvement 
Program. 

The  MPSMS  is  administered  through  the  Quality  Improvement  Group  (QIG) 
in  the  Office  of  Clinical  Standards  and  Quality  (OCSQ)  at  CMS.  The  project  goal 
contributes  to  a  larger  mission  of  accelerating  the  pace  of  health  care  quality 
improvement  in  the  United  States.  In  this  context,  the  QIG  subscribes  to  the 
principle  explicitly  stated  in  the  2003  IOM  report  on  Patient  Safety  Data 
Standards  “that  patient  safety  is  indistinguishable  from  the  delivery  of  quality 
care.”4  Within  that  larger  mission,  the  role  of  the  MPSMS  is  only  the 
measurement  of  adverse  events.  The  project  has  no  capacity  to  initiate 
interventions  or  affect  change  in  the  topics  observed.  That  capacity  is  the  work  of 
other  quality  improvement  projects  within  QIG.  Specialization  of  MPSMS 
functionality  to  focus  only  on  measurement  significantly  reduces  the  operational 
complexity. 

The  concept  of  patient  safety  is  central  to  any  discussion  of  the  MPSMS. 
Webster’s  New  World  Dictionary  defines  safety  as  “the  condition  of  being  free 
from  harm,  injury,  or  loss.”  Thus,  patient  safety  is  the  condition  or  act  of  freeing  a 
patient  from  unintended  harm,  injury,  or  loss  that  is  more  likely  associated  with 
his/her  interaction  with  the  health  care  delivery  system  than  from  an  attendant 
disease  process. 

In  this  definition,  the  operative  words  are  “patient”  and  “harm.”  4  Not 
addressed  is  the  concept  of  error — and  this  is  an  important  distinction.  This  focus 
on  patient  harm,  with  limited  consideration  of  mitigating  factors,  is  our  definition 
of  patient-centered:  the  well-being  and  the  perspective  of  the  patient  remain  the 
primary  consideration.  A  patient-centered — as  opposed  to  a  provider-centered — 
approach  is  essential  to  our  quality  improvement  efforts  at  CMS.  By  placing  our 
focus  on  the  protection  of  the  patient,  CMS  engages  a  more  achievable  goal  to 
safety  design — fault  tolerance.  Fault  tolerance  is  the  capacity  of  a  system,  in  this 
case  a  health  care  system,  to  gracefully  recover  from  an  unexpected  failure  before 
the  typical  consequence  of  that  failure  can  adversely  affect  the  desired  result.  As 
opposed  to  a  provider-centered  focus  of  error  elimination,  fault  tolerance  attempts 
to  create  systems  that  minimize  or  eliminate  patient  harm  despite  the  introduction 
of  errors.  There  is  ample  precedent  for  this  approach.  Fault  tolerance  is  widely 
accepted  by  the  broader  safety  community  in  other  disciplines  such  as  airline 
safety  and  computer  network  reliability.  These  areas  recognize  that  the  complete 
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elimination  of  error  in  human  endeavor  is  not  likely  to  succeed  under  any 
conditions.  In  On  Hierarchical  Design  of  Computer  Systems  for  Critical 
Applications,  Peter  Gabriel  Neumann  states  very  clearly:  “No  system  is 
guaranteed  to  work  properly  all  the  time. .  .humans  in  the  loop  may  add  to  the 
problem  rather  than  improve  it.”5  Patient-centered  intent  carries  huge  practical 
implications  in  the  conduct  of  a  health  care  surveillance  system,  one  of  the  first 
and  most  notable  of  which  is  how  MPSMS  addresses  the  question  of  comorbid 
risk.  Robust  patient  risk  stratification  is  an  important  feature  of  systems  designed 
to  draw  comparisons  of  provider  perfonnance.  Risk  stratification  recognizes  that 
medical  outcomes  are,  in  part,  influenced  by  the  comorbid  attributes  of  the 
patients  involved.  But  when  the  outcome  studied  is  hann,  the  results  are  typically 
presented  in  such  a  way  as  to  dissipate  some  provider  culpability  by  establishing  a 
relative  ranking  of  error.  The  result  is  that  patients  with  greater  comorbid  risk  are 
considered  less  harmed  (e.g.,  a  pneumothorax  subsequent  to  subclavian 
catheterization  is  not  considered  as  egregious  in  the  morbidly  obese).  This  is 
inherently  a  provider-centered  approach,  ill-suited  for  a  national  surveillance 
system  aimed  only  at  estimating  patient  harm.  While  the  MPSMS  has  no  mandate 
or  capacity  to  measure  or  publicly  report  provider  rates  of  adverse  events,  our 
fiduciary  responsibility  toward  public  disclosure  extends  well  into  the  CMS 
mission  as  a  piece  of  the  national  public  health  infrastructure. 

Part  of  the  CMS  public  health  mission  has  always  been  the  intent  to  assure 
that  the  MPSMS  remains  applicable  beyond  a  guide  for  quality  improvement 
projects  at  CMS.  Another  goal  has  been  our  inclusion  in  the  annual  National 
Healthcare  Quality  Report.  MPSMS  modules  may  be  applied  by  any  entity  with 
access  to  a  large  chart-based  data  source.  Thus  in  2005  the  CMS  will  make 
available  MPSMS  modules  to  hospitals,  provider  networks,  and  health  plans  to 
aid  in  their  internal  estimation  of  adverse  events. 

Relevance 

The  CMS  fiduciary  responsibility  has  direct  implications  on  the  topics 
considered  in  the  MPSMS.  As  a  federal  agency,  CMS  must  keep  its  patient  safety 
policy  consistent  with  the  distinct  public  health  mission  of  the  Department  of 
Health  and  Human  Services  (DHHS). 

Unlike  other  adverse  event  surveillance  projects,  the  MPSMS  uses  explicit 
chart  review.  This  approach  requires  that  the  system  maintain  a  discrete,  well- 
defined  list  of  surveillance  topics.  Compared  with  other  patient  safety  projects 
that  attempt  to  identify  all  adverse  events  in  a  given  care  episode,2  the  MPSMS 
only  has  the  capacity  to  identify  rates  for  select  adverse  events.  For  the  purposes 
of  an  explicit  review  program,  the  MPSMS  must  address  the  question  of  which 
discrete  topics  to  consider;  in  other  words,  which  topics  are  relevant.  Our  current 
topics  have  largely  been  selected  based  on  the  prioritized  list  of  patient  safety 
topics  compiled  by  the  AHRQ,  the  DHHS  agency  charged  with  providing 
leadership  in  patient  safety.  The  IOM,  in  its  2000  report,  recommended  that 
AHRQ  identify  and  disseminate  best  patient  safety  practices.1  The  product  of  that 
recommendation  is  Making  Health  Care  Safer:  A  Critical  Analysis  of  Patient 
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Safety  Practices  6  Chapter  57  of  that  compendium  provides  a  list  of  patient  safety 
topics  “ranked  on  a  metric  for  potential  impact.”  Table  2  lists  the  topics  in  which 
the  MPSMS  is  currently  collecting  data  (production),  topics  for  which  data 
collection  began  in  July  2004  (pre-production),  and  the  two  topics  undergoing 
criteria  development  (developmental). 


Table  2:  MPSMS  Topics 


Production 

Pre-production 

Developmental 

Bloodstream  infections* 

Adverse  drug  events  ** 

Hospital  falls** 

CVC* 

Pressure  ulcers* 

Angiographic  adverse 
events*** 

Hip/knee  replacement 

Postoperative  cardiac 

Postoperative  pneumonia 

events* 

Methicillin-resistant 

Staphylococcus  aureus  and 

Postoperative  DVT/PE* 

Hospital  acquired  UTI** 

Ventilator  associated 
pneumonia** 

Vancomycin-resistant 

Enterococcus* 

CVC  =  central  venous  catheter;  DVT  =  deep  vein  thrombosis;  PE  =  Pulmonary  embolism;  UTI  = 
urinary  tract  infection 

*  Greatest  impact  and  strength  of  evidence 
**  High  impact  and  strength  of  evidence 

***  Contrast-induced  renal  failure,  an  angiographic  complication  is  rated  as  medium  impact  and 
strength  of  evidence.  The  other  angiographic  complication  being  considered  by  the  MPSMS, 
mechanical  hemostatic  adverse  events  was  identified  by  the  FDA  as  of  emerging  importance. 

After  independently  rating  patient  safety  topics  for  potential  impact  and  the 
strength  of  evidence  supporting  a  preventative  intervention,  AHRQ  created  a  five- 
tier  hierarchy — greatest,  high,  medium,  low,  and  lowest  impact.  By  heavily 
leveraging  our  work  in  those  topics  rated  for  greatest  or  high  impact  and  strength 
of  evidence,  the  MPSMS  maintains  a  national  public  health  relevance  (see  Table  2). 

Beyond  the  conscious  decision  to  assure  relevance  by  favoring  topics  ranked 
by  AHRQ  as  significant  patient  safety  issues,  the  MPSMS  also  seeks  out  topics 
based  on  feedback  from  other  federal  agencies  as  potentially  significant.  One 
topic,  angiographic  adverse  events,  was  initially  considered  since  incorporation  in 
MPSMS  represents  an  opportunity  to  identify  the  rate  of  adverse  events 
associated  with  the  introduction  of  an  emerging  technology,  postangiographic 
hemostatic  devices.  After  being  alerted  to  this  potential  growing  patient  safety 
concern  by  TEP  members  in  the  FDA,  the  MPSMS  team  chose  this  topic  in  an 
effort  to  be  proactive  in  the  field  of  patient  safety  surveillance.  When  considered 
with  the  established  patient  safety  topic  of  contrast-induced  renal  failure, 
angiographic  associated  adverse  events  met  our  threshold  for  consideration. 

Finally,  the  MPSMS  team  also  considers  topics  for  inclusion  that  have  a 
unique  relevance  to  CMS  (e.g.,  adverse  events  associated  with  hip/knee 
replacements  and  postoperative  pneumonia).  Hip/knee  replacements  account  for 
8.5  percent  of  major  surgical  procedures  in  the  Medicare  population — double  that 
of  the  next  largest  category,  open  heart  procedures.  This  volume  of  major  surgical 
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cases  within  our  population  makes  this  orthopedic  topic  inherently  relevant  for 
CMS.  Our  second  topic  not  rated  by  AHRQ,  postoperative  pneumonia,  is  the 
second  most  frequent  postoperative  complication  after  major  surgery.7  This 
provides  an  excellent  opportunity  to  highlight  the  MPSMS  rationale  for  topic 
exclusion.  While  we  are  examining  the  second  most  frequent  postoperative 
complication,  the  first  and  most  frequent,  postoperative  wound  infection,  was  not 
selected  as  a  topic  because  it  represents  a  unique  inpatient  surveillance 
challenge — a  challenge  that  highlights  some  advantages  and  limitations  of  our 
primary  data  source,  the  inpatient  medical  record.  The  trend  toward  shorter  acute 
care  postoperative  stays  has  shifted  the  diagnostic  location  for  a  large  percentage 
of  postoperative  wound  infections  out  of  the  inpatient  setting.  A  significant  extent 
of  surgical  site  infection  documentation  is  absent  within  the  inpatient  record. 
Without  reliable  proxies  for  this  event  in  the  administrative  claims,  our  ability  to 
identify  postoperative  wound  infection  is  limited  to  early  infection  (occurring 
during  inpatient  convalescence)  and  very  severe  infections  (those  that  require 
readmission  and/or  return  to  the  operating  room).  Alternatively,  the  typical 
clinical  history  and  diagnosis  of  postoperative  pneumonia  makes  surveillance 
through  inpatient  record  review  more  reliable,  given  the  recent  emphasis  on  early 
postoperative  discharge  (the  typical  SSI  manifests  between  7-10  days,  whereas 
postoperative  pneumonia  is  typically  discovered  in  postoperative  days  3-7). 

The  criteria  used  to  select  the  adverse  medical  event  categories  were 
developed  by  the  MPSMS  team  and  include  a  preference  for  topics  deemed 
relevant  by  the  DHHS  patient  safety  lead  agency  AHRQ.  Beyond  that,  additional 
criteria  were  developed  based  on  a  careful  review  of  the  patient  safety  literature.  It 
was  believed  that  these  criteria  would  identify  those  events  that  would  yield  both 
common  adverse  medical  events  and  hospital  processes  of  care.  Thus,  the  five 
criteria  for  selecting  adverse  medical  events  were — 

•  the  adverse  event  can  be  found.  The  adverse  event  occurrence  or  the 
occurrence,  of  an  event  considered  as  a  proxy  for  an  adverse  event,  is 
likely  to  be  documented  in  a  characteristic  fashion  and  location(s)  in 
the  medical  record. 

•  the  adverse  event  is  likely  to  be  associated  with  exposure  to  a  specific 
process  of  care. 

•  the  adverse  event  is  common.  Even  if  a  common  exposure  results  in 
many  adverse  events,  the  MPSMS  measures  only  a  particular  common 
adverse  event  associated  with  the  exposure. 

•  the  adverse  event  is  responsible  for  serious  morbidity/mortality. 

•  the  adverse  event  is  preventable  or  repairable.  Processes  can  be 
developed  or  improved  to  prevent  or  decrease  the  morbidity  resulting 
from  the  adverse  event. 

For  abstraction  efficiency  at  the  CDACs,  elements  common  to  multiple  topics 
are  integrated  within  the  software  tool.  This  grouping  of  common  elements  at  the 
CDACs  is  to  incorporate  a  topic  into  the  CD  AC  MPSMS  workflow  as  a  whole. 
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All  required  components  necessary  to  make  an  adverse  event  determination  are 
maintained  as  discrete,  self-contained  modules.  Modularity  provides  for  easy 
dissemination  of  the  MPSMS  methods  beyond  the  CMS  Quality  Improvement 
Program.  A  health  care  services  entity  may  use  any  of  the  MPSMS  production 
topic  modules  for  their  own  internal  patient  safety  review.  Central  venous 
catheters,  for  example,  apply  it  to  a  large  chart-based  data  source,  and  identify  an 
adverse  event  rate  for  that  topic.  The  explicit  review  design  provides  portability  to 
other  health  care  systems  that  implicit  review  can  not. 

Transparency 

The  third  requirement  axis  is  transparency.  At  the  MPSMS,  we  accept  the 
absolute  need  for  transparency  in  a  patient  safety  surveillance  system. 
Transparency  is  the  antithesis  of  the  most  insidious  feature  of  our  current  health 
care  culture:  secrecy.  The  patient  safety  literature  is  replete  with  studies  that 
demonstrate  that  the  current  culture  of  health  care  delivery  remains  a  potential 
barrier  to  improving  the  safety  of  our  health  care  delivery  systems.8  10 

That  culture  of  secrecy  is,  in  part,  reinforced  by  the  absence  of  a  robust, 
standardized  health  data  interchange,  taxonomy,  and  medical  knowledge 
representation.2  Ironically,  resting  at  the  center  of  this  problem  is  the  medical 
record.  Providers  have  compiled  the  medical  record,  our  primary  data  source, 
which  has  remained  largely  unchanged  for  the  past  50  years.  The  highly 
specialized,  technical  nature  of  that  documentation,  at  times  illegible  and  often 
punctuated  with  acronyms  and  eponyms,  has  actually  created  a  barrier  to  the 
standardized,  measured  assessment  of  a  core  requirement  of  health  care 
systems — namely,  “do  no  harm.”  In  the  context  of  a  patient  safety  surveillance 
system,  the  medical  record  barrier  is  actually  twofold,  presenting  two 
challenges — indexing  and  analysis.  Chart-based  patient  safety  reviews  are  done 
by  employing  clinical  experts  for  both  record-indexing  and  analysis.  This  implicit 
review  process  cultivates  the  perception  that  health  care  delivery  remains  a  closed 

3  11 

society,  opaque  even  to  external  review  for  issues  of  quality  and  safety.  ’ 

At  the  MPSMS,  we  dissociate  the  indexing  of  the  chart  (performed  by  CDAC 
abstractors)  from  the  analysis  for  adverse  events  using  the  algorithms.  But  for  the 
MPSMS,  the  concept  of  transparency  extends  beyond  the  application  of  an 
explicit  algorithmic  definition  to  a  data  source  of  potential  adverse  events.  Our 
current  design  provides  an  opportunity  for  total  disclosure  of  our  processes.  So,  in 
the  context  of  transparency,  it  is  important  to  address  the  qualitative  and 
quantitative  value  of  our  approach,  namely,  the  perceived  value  to  the  public,  the 
reliability  of  our  abstraction  process,  and  a  comparative  analysis  of  the  cost-per- 
chart  reviewed. 

As  mentioned  above,  by  fully  defining  the  properties  of  selected  adverse 
events,  the  MPSMS  has  been  able  to  dissociate  indexing  from  analysis  and  create 
an  explicit  review  process  that  is  inherently  more  transparent  than  the  previous 
reporting  systems.  Some  of  those  systems,  such  as  the  Harvard  Medical  Practice 
Review,3  are  relatively  transparent  because  they  used  standardized  reviewer 
training  and  broad  definitions  to  clarify  how  their  clinicians  were  guided  to  the 
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decisions  to  identify  adverse  events.  Even  with  this  approach,  any  implicit  review 
project  has  a  significant  level  of  opacity  to  external  review.  They  each  have  a  tacit 
analysis  element  that  is  predominantly  the  prerogative  of  an  individual  physician 
or  nurse  reviewer.  Health  care  is  ill-served  if  adverse  event  detennination  is 
perceived  to  be  only  the  purview  of  expert  clinicians.12  But  explicit  review  has 
value  beyond  the  question  of  public  perception.  As  both  Donabedian13  and  Hiatt14 
note,  determinations  made  by  applying  explicit  criteria  are  inherently  more 
reliable. 

In  addition  to  improved  inter-rater  reliability,  the  MPSMS  is  cost  effective 
when  compared  with  implicit  review  projects.  For  purposes  of  this  comparison, 
we  are  only  considering  the  average  cost-per-chart,  excluding  infrastructure  costs 
that  will  not  be  incurred  by  health  care  systems  seeking  to  use  any  of  the  MPSMS 
modules.  These  include  costs  associated  with  our  technical  expert  panels; 
algorithm  testing;  software  development;  and  chart  copying  and  shipping.  While 
these  MPSMS  infrastructure  costs  are  significant,  the  goal  of  this  cost  comparison 
is  to  present  the  MPSMS  explicit-review  standard  as  an  alternative  for  health  care 
providers  and  payers.  Our  expectation  is  not  to  provide  a  comparative  estimate  for 
providers  and  payors,  reproducing  the  MPSMS  infrastructure.  Rather,  the  analysis 
provides  a  benchmark  to  providers  and  payors  who  might  use  an  MPSMS  topic 
module  to  estimate  their  own  adverse  event  rates,  given  a  large  chart  data  source. 
As  described  earlier,  each  MPSMS  topic  is  maintained  as  a  discrete,  self- 
contained,  portable  unit.  This  portability  will  extend  to  all  critical  components  of 
our  process,  including  algorithms,  definitions,  and  training  tools.  Our  expectation 
is  that  the  MPSMS  standard  for  a  topic,  which  can  be  considered  an  explicit 
adverse  event  definition,  will  be  exported  and  applied  beyond  our  current 
Medicare  fee-for-service  population. 

We  calculated  our  implicit  review  average  cost-per-chart  using  data  obtained 
through  the  clinical  review  arm  of  MPSMS  algorithm  development  process. 
Because  the  MPSMS  beta  clinical  review  is  implicit,  we  are  able  to  compare  per 
chart  costs  with  the  costs  for  each  production  chart  reviewed  at  the  CD  AC.  On 
that  basis  we  have  found  that  implicit  clinical  review  has  an  average  cost-per- 
chart  of  $350  and  the  MPSMS  production  sample  average  cost-per-chart  is  $74. 
The  efficiencies  generated  due  to  CD  AC  abstractor  experience  and  the  combining 
of  common  abstracted  elements  in  the  production  sample  did  not  make  for  a  fair 
average  cost-per-chart  per  topic  comparison.  That  said,  we  should  report  that  in 
the  MPSMS  production  sample  we  are  examining  charts  for  eight  topics 
(bloodstream  infections;  central  venous  catheters  [CVC];  total  hip  and  total  knee 
replacement;  postoperative  pneumonia,  postoperative  venous  thromboembolic 
events;  postoperative  urinary  tract  infection;  and  ventilator  associated 
pneumonia),  while  in  the  implicit  review  arm,  charts  were  examined  for  no  more 
than  two  adverse  event  topics. 
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Conclusion 

The  Medicare  Patient  Safety  Monitoring  System  is  a  national  surveillance 
program  designed  to  identify  the  rates  of  selected  adverse  events  within  the 
Medicare  population.  Our  approach  employs  explicit,  rule-based  chart  review  to 
count  events.  The  intent  of  this  work  is  to  provide  a  tool  to  measure  the  safety  of 
hospital  care  delivery  systems.  While  the  MPSMS  has  no  capacity  to  intervene  in 
those  systems,  our  design  assumes  that  the  leverage  point  for  improvement 
interventions  is  fault  tolerance,  not  error  reduction.  To  that  end,  we  have  no 
measures  of  individual  provider  performance  or  provider  error.  By  embracing  a 
patient-centered  approach,  our  purpose  and  resources  mitigate  the  collection  of 
comorbid  risk  factors  for  each  measure.  Robust  comorbid  risk  stratification, 
useful  for  provider  comparison,  is  resource-intensive  and  of  little  use  in  a  system 
with  maximal  resolution  at  the  State  level.  The  MPSMS  maintains  relevance  by 
choosing  topics  based  on  an  overarching  agenda  to  improve  the  safety  of  our 
nation’s  hospitals.  Thus,  our  choice  of  topics  is  heavily  weighted  to  the  issues  that 
provide  the  greatest  public  health  benefit. 

The  public  benefit  would  be  diminished  if  our  methods  did  not  wholly 
conform  to  a  standard  of  transparency  from  topic  selection  through  results 
reporting.  This  transparency  is  a  natural  extension  of  our  decision  to  deviate  from 
other  large  patient  safety  surveillance  projects  and  employ  explicit  review.  The 
return  on  this  decision  is  high  inter-rater  reliability  and  a  lower  cost-per-chart 
reviewed  than  implicit  review  systems. 

The  MPSMS,  created  by  the  DHHS  Patient  Safety  Task  Force  (PSTF),  is  a 
direct  byproduct  of  the  2000  IOM  report,  To  Err  Is  Human:  Building  a  Safer 
Health  System .  Inherent  in  the  mission  of  the  PSTF — and,  by  extension,  CMS — 
is  a  duty  to  understand  the  magnitude  of  patient  adverse  events  in  our  nation.  The 
dual  fiduciary  duties  of  protecting  the  Medicare  Trust  Fund  and  improving  the 
safety  of  the  health  care  systems,  affords  CMS  access  to  the  breadth  and  depth  of 
information  necessary  to  perform  the  tasks  central  to  the  goal  of  the  MPSMS. 

As  to  the  MPSMS,  our  mandate  from  the  DHHS,  our  unparalleled  access  to 
chart-level  information,  and  our  goal  of  keeping  the  patient  at  the  center  of  our 
work,  all  require  that  this  opportunity  to  assess  the  safety  of  our  nation’s  health 
care  delivery  systems  reaches  beyond  the  limitations  of  identifying  provider  error 
by  implicit  review.  In  the  MPSMS,  we  have  changed  the  patient  safety  assessment 
paradigm  to  employ  explicit  review  to  identify  patient  harm.  But  even  that  unique 
singularity  of  purpose  must  be  governed  by  requirements  that  meet  both  the  spirit 
of  the  IOM  call  for  a  new  era  in  patient  safety  and  federal  programmatic 
practicability.  By  declaring  and  maintaining  our  position  relative  to  the  three 
requirement  planes  of  intent,  relevance,  and  transparency,  the  MPSMS  meets  both 
governing  principles. 
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